Capturing fine-grained details for video-based automation of suturing skills assessment.

OBJECTIVES: Manually-collected suturing technical skill scores are strong predictors of continence recovery after robotic radical prostatectomy. Herein, we automate suturing technical skill scoring through computer vision (CV) methods as a scalable method to provide feedback. METHODS: Twenty-two surgeons completed a suturing exercise three times on the Mimic™ Flex VR simulator. Instrument kinematic data (XYZ coordinates of each instrument and pose) were captured at 30 Hz. After standardized training, three human raters manually video segmented suturing task into four sub-stitch phases (Needle handling, Needle targeting, Needle driving, Needle withdrawal) and labeled the corresponding technical skill domains (Needle positioning, Needle entry, Needle driving, and Needle withdrawal). The CV framework extracted RGB features and optical flow frames using a pre-trained AlexNet. Additional CV strategies including auxiliary supervision (using kinematic data during training only) and attention mechanisms were implemented to improve performance. RESULTS: This study included data from 15 expert surgeons (median caseload 300 [IQR 165-750]) and 7 training surgeons (0 [IQR 0-8]). In all, 226 virtual sutures were captured. Automated assessments for Needle positioning performed best with the simplest approach (1 s video; AUC 0.749). Remaining skill domains exhibited improvements with the implementation of auxiliary supervision and attention mechanisms when deployed separately (AUC 0.604-0.794). All techniques combined produced the best performance, particularly for Needle driving and Needle withdrawal (AUC 0.959 and 0.879, respectively). CONCLUSIONS: This study demonstrated the best performance of automated suturing technical skills assessment to date using advanced CV techniques. Future work will determine if a "human in the loop" is necessary to verify surgeon evaluations.

Assessment of Minimally Invasive Suturing Skills: Is Instrument Tracking an Accurate Prediction?

Introduction: Minimally invasive surgery (MIS) suturing demands advanced surgical skills. Therefore, it is important these skills are adequately trained and assessed. Assessment and feedback can consist of judgments and scores of expert observers or objective parameters using instrument tracking. The aim of this study was to determine to what extent objective parameters correspond to expert assessment. Methods: Participants performed an intracorporeal suturing task on the EoSim simulator repeatedly (maximum 20 repetitions) during training. The best discriminating parameters, which previously shown construct validation, were combined into a composite score, using regression analysis. All videos were blinded and assessed by 2 independent reviewers using the validated laparoscopic suturing competency assessment tool (LS-CAT). These scores were compared with the composite score. Results: A 100 videos of 16 trainees, during separate points on their learning curve, and 8 experts were used. The parameters "time" and "distance" were statistically significantly correlated with all LS-CAT domains. The composite score (calculated from "time" and "distance") showed improvement between the first and the last knot (57% versus 94%, P < .001). Also the LS-CAT score improved (28 versus 17, P < .001). However, the correlation of the composite score with the LS-CAT score was weak (R: 0.351), with an accuracy of 55/100 when pooling the outcomes based on inadequate, adequate, or good performance. Conclusion: Instrument tracking parameters (using Surgtrac) could give an indication of the skill level, however, it missed important elements, essential for reliable assessment. Therefore, expert assessment remains superior to determine the skill level in MIS suturing skills.

Developing artificial intelligence models for medical student suturing and knot-tying video-based assessment and coaching.

BACKGROUND: Early introduction and distributed learning have been shown to improve student comfort with basic requisite suturing skills. The need for more frequent and directed feedback, however, remains an enduring concern for both remote and in-person training. A previous in-person curriculum for our second-year medical students transitioning to clerkships was adapted to an at-home video-based assessment model due to the social distancing implications of COVID-19. We aimed to develop an Artificial Intelligence (AI) model to perform video-based assessment. METHODS: Second-year medical students were asked to submit a video of a simple interrupted knot on a penrose drain with instrument tying technique after self-training to proficiency. Proficiency was defined as performing the task under two minutes with no critical errors. All the videos were first manually rated with a pass-fail rating and then subsequently underwent task segmentation. We developed and trained two AI models based on convolutional neural networks to identify errors (instrument holding and knot-tying) and provide automated ratings. RESULTS: A total of 229 medical student videos were reviewed (150 pass, 79 fail). Of those who failed, the critical error distribution was 15 knot-tying, 47 instrument-holding, and 17 multiple. A total of 216 videos were used to train the models after excluding the low-quality videos. A k-fold cross-validation (k = 10) was used. The accuracy of the instrument holding model was 89% with an F-1 score of 74%. For the knot-tying model, the accuracy was 91% with an F-1 score of 54%. CONCLUSIONS: Medical students require assessment and directed feedback to better acquire surgical skill, but this is often time-consuming and inadequately done. AI techniques can instead be employed to perform automated surgical video analysis. Future work will optimize the current model to identify discrete errors in order to supplement video-based rating with specific feedback.

Automatic assessment of laparoscopic surgical skill competence based on motion metrics.

The purpose of this study was to characterize the motion features of surgical devices associated with laparoscopic surgical competency and build an automatic skill-credential system in porcine cadaver organ simulation training. Participants performed tissue dissection around the aorta, dividing vascular pedicles after applying Hem-o-lok (tissue dissection task) and parenchymal closure of the kidney (suturing task). Movements of surgical devices were tracked by a motion capture (Mocap) system, and Mocap-metrics were compared according to the level of surgical experience (experts: ≥50 laparoscopic surgeries, intermediates: 10-49, novices: 0-9), using the Kruskal-Wallis test and principal component analysis (PCA). Three machine-learning algorithms: support vector machine (SVM), PCA-SVM, and gradient boosting decision tree (GBDT), were utilized for discrimination of the surgical experience level. The accuracy of each model was evaluated by nested and repeated k-fold cross-validation. A total of 32 experts, 18 intermediates, and 20 novices participated in the present study. PCA revealed that efficiency-related metrics (e.g., path length) significantly contributed to PC 1 in both tasks. Regarding PC 2, speed-related metrics (e.g., velocity, acceleration, jerk) of right-hand devices largely contributed to the tissue dissection task, while those of left-hand devices did in the suturing task. Regarding the three-group discrimination, in the tissue dissection task, the GBDT method was superior to the other methods (median accuracy: 68.6%). In the suturing task, SVM and PCA-SVM methods were superior to the GBDT method (57.4 and 58.4%, respectively). Regarding the two-group discrimination (experts vs. intermediates/novices), the GBDT method resulted in a median accuracy of 72.9% in the tissue dissection task, and, in the suturing task, the PCA-SVM method resulted in a median accuracy of 69.2%. Overall, the mocap-based credential system using machine-learning classifiers provides a correct judgment rate of around 70% (two-group discrimination). Together with motion analysis and wet-lab training, simulation training could be a practical method for objectively assessing the surgical competence of trainees.

Assessment of Surgeon Performance of Advanced Open Surgical Skills Using a Microskills-Based Novel Curriculum.

Importance: The increase in minimally invasive surgical procedures has eroded exposure of general surgery residents to open operations. High-fidelity simulation, together with deliberate instruction, is needed for advanced open surgical skill (AOSS) development. Objective: To collect validity evidence for AOSS tools to support a shared model for instruction. Design, Setting, and Participants: This prospective cohort study included postresidency surgeons (PRSs) and second-year general surgery residents (R2s) at a single academic medical center who completed simulated tasks taught within the AOSS curriculum between June 1 and October 31, 2021. Exposures: The AOSS curriculum includes 6 fine-suture and needle handling tasks, including deep suture tying (with and without needles) and continuous suturing using the pitch-and-catch and push-push-pull techniques (both superficial and deep). Teaching and assessment are based on specific microskills using a 3-dimensional printed iliac fossa model. Main Outcomes and Measures: The PRS group was timed and scored (5-point Likert scale) on 10 repetitions of each task. Six months after receiving instruction on the AOSS tasks, the R2 group was similarly timed and scored. Results: The PRS group included 14 surgeons (11 male [79%]; 8 [57%] attending surgeons) who completed the simulation; the R2 group, 9 surgeons (5 female [55%]) who completed the simulation. Score and time variability were greater for the R2s compared with the PRSs for all tasks. The R2s scored lower and took longer on (1) deep pitch-and-catch suturing (69% of maximum points for a mean [SD] of 142.0 [31.7] seconds vs 77% for a mean [SD] of 95.9 [29.4] seconds) and deep push-push-pull suturing (63% of maximum points for a mean [SD] of 284.0 [72.9] seconds vs 85% for a mean [SD] of 141.4 [29.1] seconds) relative to the corresponding superficial tasks; (2) suture tying with a needle vs suture tying without a needle (74% of maximum points for a mean [SD] of 64.6 [19.8] seconds vs 90% for a mean [SD] of 54.4 [15.6] seconds); and (3) the deep push-push-pull vs pitch-and-catch techniques (63% of maximum points for a mean [SD] of 284.0 [72.9] seconds vs 69% of maximum points for a mean [SD] of 142.0 [31.7] seconds). For the PRS group, time was negatively associated with score for the 3 hardest tasks: superficial push-push-pull (ρ = 0.60; P = .02), deep pitch-and-catch (ρ = 0.73; P = .003), and deep push-push-pull (ρ = 0.81; P < .001). For the R2 group, time was negatively associated with score for the 2 easiest tasks: suture tying without a needle (ρ = 0.78; P = .01) and superficial pitch-and-catch (ρ = 0.79; P = .01). Conclusions and Relevance: The findings of this cohort study offer validity evidence for a novel AOSS curriculum; reveal differential difficulty of tasks that can be attributed to specific microskills; and suggest that position on the surgical learning curve may dictate the association between competency and speed. Together these findings suggest specific, actionable opportunities to guide instruction of AOSS, including which microskills to focus on, when individual rehearsal vs guided instruction is more appropriate, and when to focus on speed.

Pilot Evaluation of a Novel, Low-Cost, Simulation Model for Training and Assessment of Laparoscopic Intracorporeal Continuous Suturing.

Background: Laparoscopic intracorporeal continuous suturing is being employed in a growing number of minimally invasive procedures. However, there is a lack of adequate bench models for gaining proficiency in this complex task. The purpose of this study was to assess a novel simulation model for running suture. Methods: Participants were grouped as novice (LSN) or expert (LSE) at laparoscopic suturing based on prior experience and training level. A novel low-cost bench model was developed to simulate laparoscopic intracorporeal continuous closure of a defect. The primary outcome measured was time taken to complete the task. Videos were scored by independent raters for Global Operative Assessment of Laparoscopic Skills (GOALS). Results: Sixteen subjects (7 LSE and 9 LSN) participated in this study. LSE completed the task significantly faster than LSN (430 ± 107 vs 637 ± 164 seconds, P ≤ .05). LSN scored higher on accuracy penalties than LSE (Median 30 vs 0, P ≤ .05). Mean GOALS score was significantly different between the 2 groups (LSE 20.64 ± 2.64 vs LSN 14.28 ± 1.94, P < .001) with good inter-rater reliability (ICC ≥ .823). An aggregate score using the formula: Performance Score = 1200-time(sec)-(accuracy penalties x 10) was significantly different between groups with a mean score of 741 ± 141 for LSE vs 285 ± 167 for LSN (P < .001). Conclusion A novel bench model for laparoscopic continuous suturing was able to significantly discriminate between laparoscopic experts and novices. This low-cost model may be useful for both training and assessment of laparoscopic continuous suturing proficiency.

A new basic thoracoscopic surgical skill training and assessment system using automatic scoring techniques.

PURPOSE: We report a new thoracoscopic surgical skill training and assessment system with automatic scoring techniques, the Huaxi Intelligent Thoracoscopic Skill Training and Assessment (HITSTA) system. We also evaluated the discriminative ability of this system compared to our conventional scoring method at our institution. METHODS: We retrospectively collected training data of thoracic board-certified thoracic surgeons at West China Hospital, Sichuan University from January 1, 2018 to January 1, 2019. Surgeons were assessed by HITSTA system and human examiners simultaneously. Total scores were summed from 3 tasks (grasping with delivery, pattern cutting, and suture with knot). Bland-Altman analysis was used to test agreement of scores made by HITSTA system (automatic scoring) and human examiners (manual scoring). Differentiation ability was also compared between the two scoring methods. RESULTS: Thirty-nine surgeons were recruited. Scores made by HITSTA system and human examiners were not consistent. For suture with knot, automatic scoring method could detect the score differences between different training status (trained: 26.92 ± 12.04, untrained: 19.85 ± 11.12; p = 0.026) and training duration (< 10 h: 20.67 ± 15.23, ≥ 10 h: 31.92 ± 5.56; p = 0.003). For total scores, automatic scoring approach could discriminate between different training status (trained: 71.90 ± 12.63; untrained: 61.41 ± 13.87; p = 0.016) and training duration (< 10 h: 65.23 ± 15.31; ≥ 10 h 77.23 ± 6.94; p = 0.046). CONCLUSION: HITSTA system could discriminate the different levels of thoracoscopic surgical skills better than the traditional manual scoring method. Larger prospective studies are warranted to validate the differentiation ability of HITSTA system.

Objective assessment of intra-oral suturing for certification of competency.

INTRODUCTION: Suturing is an essential skill for dental surgeons, yet there is limited evidence for objective measures of competence in this field, which can be applied to licensing or certification processes for dental surgeons to ensure patient safety. METHODS: In an objective structured clinical examination (OSCE), participants placed two simple interrupted sutures across a lower molar socket in different suturing environments: [A] commercial simulator, [B] suturing sponge mounted in a 3D printed model, [C] suturing sponge and [D] pig's jaw. Simulators [A] and [B] were mounted into phantom heads in dental chairs, and [C] and [D] were placed on bench tops. Performance indicators, overall task success, quality of sutures placed, length of suture remaining and time taken, were analysed against participants' characteristics such as previous suturing experience. RESULTS: Participants (n = 38) took significantly longer to successfully complete the sutures on models mounted in a phantom head. Time points at which each model differentiated between competent (n = 11) and novice (n = 27) participants were as follows: model A (3:26 min), B (2:41), C (2:05) and D (2:06). Model A showed a significant association between task completion and previous suturing experience (p = .006). CONCLUSION: Suturing simulators can be used to objectively differentiate between novice and competent dental surgeons. Model A demonstrated construct validity, as it reliably identified competence using time and task completion as objectively measurable parameters. Utilising model A, the candidate of unknown ability could be identified as appropriately skilled at intra-oral suturing. To ensure patient safety, further research is needed to identify other objective measures of competence in simulation environments.

Randomized controlled trial comparing the effectiveness of mass and spaced learning in microsurgical procedures using computer aided assessment.

Spaced-learning refers to teaching spread over time, compared to mass-learning where the same duration of teaching is completed in one session. Our hypothesis is that spaced-learning is better than mass-learning in retaining microsurgical suturing skills. Medical students were randomized into mass-learning (single 8-h session) and spaced-learning (2-h weekly sessions over 4 weeks) groups. They were taught to place 9 sutures in a 4 mm-wide elastic strip. The primary outcome was precision of suture placement during a test conducted 1 month after completion of sessions. Secondary outcomes were time taken, cumulative performance, and participant satisfaction. 42 students (24 in the mass-learning group; 18 in spaced-learning group) participated. 3 students in the spaced-learning group were later excluded as they did not complete all sessions. Both groups had comparable baseline suturing skills but at 1 month after completion of teaching, the total score for suture placement were higher in spaced-learning group (27.63 vs 31.60,p = 0.04). There was no statistical difference for duration and satisfaction in either group. Both groups showed an improvement in technical performance over the sessions, but this did not differ between both groups. Microsurgical courses are often conducted in mass-learning format so spaced learning offers an alternative that enhances retention of complex surgical skills.

Motion analysis of the JHU-ISI Gesture and Skill Assessment Working Set using Robotics Video and Motion Assessment Software.

PURPOSE: The JIGSAWS dataset is a fixed dataset of robot-assisted surgery kinematic data used to develop predictive models of skill. The purpose of this study is to analyze the relationships of self-defined skill level with global rating scale scores and kinematic data (time, path length and movements) from three exercises (suturing, knot-tying and needle passing) (right and left hands) in the JIGSAWS dataset. METHODS: Global rating scale scores are reported in the JIGSAWS dataset and kinematic data were calculated using ROVIMAS software. Self-defined skill levels are in the dataset (novice, intermediate, expert). Correlation coefficients (global rating scale-skill level and global rating scale-kinematic parameters) were calculated. Kinematic parameters were compared among skill levels. RESULTS: Global rating scale scores correlated with skill in the knot-tying exercise (r = 0.55, p = 0.0005). In the suturing exercise, time, path length (left) and movements (left) were significantly different (p < 0.05) for novices and experts. For knot-tying, time, path length (right and left) and movements (right) differed significantly for novices and experts. For needle passing, no kinematic parameter was significantly different comparing novices and experts. The only kinematic parameter that correlated with global rating scale scores is time in the knot-tying exercise. CONCLUSION: Global rating scale scores weakly correlate with skill level and kinematic parameters. The ability of kinematic parameters to differentiate among self-defined skill levels is inconsistent. Additional data are needed to enhance the dataset and facilitate subset analyses and future model development.

Development of a Low-cost, High-fidelity Skin Model for Suturing.

BACKGROUND: In a survey of students at our institution, suturing was the most desired workshop for simulation; however, cost, quality, and availability of skin pads is often prohibitive for suturing workshops. In-hospital fabrication may be utilized to manufacture noncommercial, high-fidelity, and low-cost simulation models. We describe the production, value, and face validation of our simulated skin model. MATERIALS AND METHODS: Using an in-hospital fabrication laboratory, we have developed a model for skin and subcutaneous tissue. Our model uses a variety of commercially available materials to simulate the epidermis, dermis, subcutaneous fat, fascia, and muscle. A cost analysis was performed by comparing it with other commonly used commercial skin models. Expert surgeons assessed the material characteristics, durability, and overall quality of our model in comparison with other commercial models. RESULTS: The materials cost of our novel skin pad model was 30.9% of the mean cost of five different commonly used foam and silicone-based commercial skin models. This low-cost model is more durable than the commercial models, does not require skin pad holders, and is of higher fidelity than the commercial products. In addition to skin closure, our model may be used to simulate fascial closure or fasciotomy. CONCLUSIONS: Model creation using in-hospital workspaces is an effective strategy to decrease cost while improving quality of surgical simulation. Our methods for creation of an inexpensive and high-fidelity skin pad may be purposed for several soft tissue models.

Construct validity of the SurgForce system for objective assessment of laparoscopic suturing skills.

BACKGROUND: Laparoscopic surgery requires a new set of skill to be learned by the surgeons, of which the most relevant is tissue manipulation. Excessive forces applied to the tissue can cause rupture during manipulation or ischemia when confronting both sides of the tissue. The aim of this study is to establish the construct validity of the SurgForce system for objective assessment of advanced laparoscopic skills, based on the force signal generated during suture tasks, and the development of force parameters for evaluating tissue handling interaction. METHODS: The SurgForce system, a tissue handling training device that measures dynamic force, was used to capture the force generated by surgeons with different levels of laparoscopic experience. For construct validity, 37 participants were enrolled in this study: 19 medical students, 12 residents of surgical specialties and 6 expert surgeons. All participants performed an intracorporeal knotting suture task over a synthetic tissue pad with a laparoscopic box-trainer. The force performance of the participants was analyzed using 11 force-based parameters with the application of the SurgForce system. Statistical analysis was performed between novice, intermediate, and expert groups using a Kruskal-Wallis test, and between the pairs of groups using a Mann-Whitney U-test. RESULTS: Overall, 9 of the 11 force-related parameters showed significant differences between the three study groups. Results between the pairs of groups presented significant differences in 5 force parameters proposed. Construct validity results demonstrated that the SurgForce system was able to differentiate force performance between surgeons with different levels of laparoscopic experience. CONCLUSION: The SurgForce system was successfully validated. This force system showed its potential to measure the force exerted on tissue for objective assessment of tissue handling skills in suturing tasks. Furthermore, its compact design allows the use of this device in conventional laparoscopic box-trainers.

Experimental Models of Partial Intestinal Obstruction in Young Mice: Establishment, and Evaluation.

BACKGROUND: Partial small bowel obstruction (SBO) is a common, potentially hazardous, surgical entity caused by numerous factors in humans. A number of techniques have been reported as efficient to simulate partial SBO in murine models. However, there is little data concerning their long-term survival. Our study presents a novel technique and evaluates its long-term efficiency compared with other commonly used techniques. MATERIALS AND METHODS: Sixty C57BL/6 mice aged 6 to 8 wk were randomly divided into five intervention groups: ligation, intestinal ring, partial ligation, microclips, and the novel triple suture technique. The ring groups were subdivided into narrow, medium, and wide ring and partial ligation groups were subdivided at 1/3, 1/2, and 2/3 of the lumen. Survival cutoff time was set at 4 wk. Animals were then euthanized and small bowel muscle layer thickness was histopathologically evaluated. RESULTS: None of the animals of the ligation and the ring groups reached the cutoff survival time. The mortality rate of the partial ligation and the microclips groups at the 4-week period were 33.3% and 0%, respectively. However, elimination of the performed intervention was revealed at the time of euthanasia and no alterations of the muscle layer were revealed at histopathology. The "triple suture" group had a survival rate of 90% until euthanasia and the sutures were apparent in all cases. Macroscopic evaluation showed small to mild proximal lumen dilatation in 6 of 10 animals. Histopathological evaluation of the specimens confirmed the partial obstruction. CONCLUSIONS: The "triple suture" technique is a new, robust, reliable, and inexpensive technique for experimental long-standing partial SBO, with very low mortality.

Using Low-Cost Models for Training First-Year Pediatric Residents on 4 Accreditation Council for Graduate Medical Education-Required Procedures: A Pilot Study.

OBJECTIVES: The aims of the study were to assess the feasibility of using low-cost models to train first-year pediatric residents and to examine whether residents who receive such training will be as competent as their experienced colleagues in performing 4 American College of Graduate Education-required procedures, including suturing, splinting, lumbar puncture, and venipuncture. METHODS: We performed a pilot study with postgraduate year (PGY) 1 to 3 residents. Postgraduate year 1 residents completed a self-assessment questionnaire before the onset of training. A lecture was given to all PGY levels residents about procedural techniques. The PGY-1 residents practiced these techniques on low-fidelity models immediately after the lecture. One and 9 months after the initial lecture, all residents were assessed on these models using a 10-point checklist for each skill. RESULTS: Thirteen PGY-1 residents, 10 PGY-2 residents, and 10 PGY-3 residents completed the study. There was no statistically significant difference in performance of PGY-1 residents when compared with PGY-2 and PGY-3 residents in performing lumbar puncture, venipuncture, and suturing on models in the initial assessment that was performed 1 month after the lecture. Postgraduate year 1 residents performed equally well to PGY-3 residents and significantly (P < 0.05) better than PGY-2 residents, in splinting.There was no statistically significant difference between groups at final follow-up, supporting that training on models could help enhance proficiency among residents. CONCLUSIONS: This pilot study supports the feasibility of using low-cost models to train residents on invasive and painful procedures. Furthermore, residents trained on models showed maintenance of skills for a 9-month period.

The impact of rater training on the psychometric properties of standardized surgical skill assessment tools.

INTRODUCTION: Competency-based frameworks are common in surgical training. However, the optimal use of standardized technical assessments is not well defined. We investigated the effect of rater training (RT) on the reliability and validity of four assessment tools. MATERIALS AND METHODS: Forty-Seven surgeons were randomized to RT (N = 24) and no training (N = 23) groups. A task-specific checklist, pass-fail, visual analog, and OSATS global rating scale (GRS) were used to assess trainee knot-tying and suturing tasks. Delayed assessment was performed two weeks later. Internal consistency, intra/inter-rater reliability, and construct validity were measured. RESULTS: The GRS had superior reliability and validity compared to the other tools regardless of training. No significant differences between training groups was found. However, the RT group trended to improved reliability for all tools at both assessments. CONCLUSIONS: RT did not lead to significant improvements in skills assessments. Standardized assessments (OSATS GRS) are preferred due to their superior reliability and validity over other methods. Despite findings, we believe more effective training methods or repeated sessions may be required for sustained and significant effects.

Competency assessment tool for laparoscopic suturing: development and reliability evaluation.

BACKGROUND: Laparoscopic suturing can be technically challenging and requires extensive training to achieve competency. To date no specific and objective assessment method for laparoscopic suturing and knot tying is available that can guide training and monitor performance in these complex surgical skills. In this study we aimed to develop a laparoscopic suturing competency assessment tool (LS-CAT) and assess its inter-observer reliability. METHODS: We developed a bespoke CAT tool for laparoscopic suturing through a structured, mixed methodology approach, overseen by a steering committee with experience in developing surgical assessment tools. A wide Delphi consultation with over twelve experts in laparoscopic surgery guided the development stages of the tool. Following, subjects with different levels of laparoscopic expertise were included to evaluate this tool, using a simulated laparoscopic suturing task which involved placing of two surgical knots. A research assistant video recorded and anonymised each performance. Two blinded expert surgeons assessed the anonymised videos using the developed LS-CAT. The LS-CAT scores of the two experts were compared to assess the inter-observer reliability. Lastly, we compared the subjects' LS-CAT performance scores at the beginning and end of their learning curve. RESULTS: This study evaluated a novel LS-CAT performance tool, comprising of four tasks. Thirty-six complete videos were analysed and evaluated with the LS-CAT, of which the scores demonstrated excellent inter-observer reliability. Cohen's Kappa analysis revealed good to excellent levels of agreement for almost all tasks of both instrument handling and tissue handling (0.87; 0.77; 0.75; 0.86; 0.85, all with p < 0.001). Subjects performed significantly better at the end of their learning curve compared to their first attempt for all LS-CAT items (all with p < 0.001). CONCLUSIONS: We developed the LS-CAT, which is a laparoscopic suturing grading matrix, with excellent inter-rater reliability and to discriminate between experience levels. This LS-CAT has a potential for wider use to objectively assess laparoscopic suturing skills.

Optimizing resource utilization during proficiency-based training of suturing skills in medical students: a randomized controlled trial of faculty-led, peer tutor-led, and holography-augmented methods of teaching.

BACKGROUND: Suturing is a fundamental skill in undergraduate medical education. It can be taught by faculty-led, peer tutor-led, and holography-augmented methods; however, the most educationally effective and cost-efficient method for proficiency-based teaching of suturing is yet to be determined. METHODS: We conducted a randomized controlled trial comparing faculty-led, peer tutor-led, and holography-augmented proficiency-based suturing training in pre-clerkship medical students. Holography-augmented training provided holographic, voice-controlled instructional material. Technical skill was assessed using hand motion analysis every ten sutures and used to construct learning curves. Proficiency was defined by one standard deviation within average faculty surgeon performance. Intervention arms were compared using one-way ANOVA of the number of sutures placed, full-length sutures used, time to proficiency, and incremental costs incurred. Surveys were used to evaluate participant preferences. RESULTS: Forty-four students were randomized to the faculty-led (n = 16), peer tutor-led (n = 14), and holography-augmented (n = 14) intervention arms. At proficiency, there were no differences between groups in the number of sutures placed, full-length sutures used, and time to achieve proficiency. The incremental costs of the holography-augmented method were greater than faculty-led and peer tutor-led instruction ($247.00 ± $12.05, p < 0.001) due to the high cost of the equipment. Faculty-led teaching was the most preferred method (78.0%), while holography-augmented was the least preferred (0%). 90.6% of students reported high confidence in performing simple interrupted sutures, which did not differ between intervention arms (faculty-led 100.0%, peer tutor-led 90.0%, holography-augmented 83.3%, p = 0.409). 93.8% of students felt the program should be offered in the future. CONCLUSION: Faculty-led and peer tutor-led instructional methods of proficiency-based suturing teaching were superior to holography-augmented method with respect to costs and participants' preferences despite being educationally equivalent.

Novel quantitative and objective structured assessment of technical skill for slip knotting.

OBJECTIVE: Currently, no quantitative and objective method has been established for evaluating competencies in basic surgical techniques. The aim of this study was to develop a structured assessment tool for slip knotting and verify how well current board certification system discriminates the level of basic surgical skill. METHODS: We examined 171 cardiovascular surgical fellows using a novel assessment method for slip knotting that was developed by the committee of the Under-Forty of the Japanese Society of Cardiovascular Surgery. We compared the scores and examinees' surgical experience for validation. We analyzed the relationship between board certification and the scores. RESULTS: The scores differentiated the general surgical board-certified surgeons from those without certification. Surgical experiences such as training years and number of operated cases and scores were correlated. Among the board-certified surgeons, the group with daily off-the-job training, or simulator-based skill training had a significantly higher mean score (67.4 ± 3.0 vs 55.4 ± 3.1, p = 0.008) and lower rate of poor scorers (7.1% vs 38.5%, p = 0.004). A multivariate analysis revealed that board certification did not predict high scores. Daily off-the-job training was the only independent predictor of high scores (odds ratio: 2.41, 95% confidence interval: 0.01-1.20, p = 0.014). CONCLUSIONS: This novel quantitative and objective assessment tool for technical skill in slip knotting was found to be valid to examine the skill for slip knotting. In this study, current board certification discriminated the level of basic surgical skill. However, it could not distinguish extremely low scorers perfectly. Some board-certified surgeons showed poor technical competency, especially those without off-the-job training.

A Simulation Suture Compared With a Clinical Suture for Training on Laparoscopic Simulators: Objective Measurements Indicate Acceptable Physical Characteristics.

OBJECTIVE: Simulation sutures are a low-cost option for training purposes, but they may not perform as well as the more expensive clinical suture. Trainees at our institution have raised concerns about their quality and integrity compared with clinical suture. The objective was to determine whether significant differences in strength of the sutures and knot holding capabilities between low and high-cost sutures existed. METHODS: Two sutures were compared: 3-0 braided silk simulation suture (Sim*Vivo LLC, Willsboro, NY) and 3-0 Perma-Hand silk braided clinical suture (Ethicon, Somerville, NJ). The diameter of the suture was assessed with light microscopy. Tensile strength of the suture and knotted suture were assessed. Both sutures were tested dry and wet. RESULTS: Tensile strength of the sutures, knotted or unknotted, were not significantly different. Knot type did not change this comparison. Soaking in saline did not change the tensile strength but did introduce a substantial difference in knot failure mode between sutures. The mean diameter of the Ethicon suture was larger than that of the Sim*Vivo suture, which could influence suture behavior. CONCLUSIONS: The applied mechanical tests identified that clinical and simulation sutures differ some in their inherent mechanical characteristics related to suture handling. However, these differences did not translate to a key measure of performance of a sutured junction, namely, the strength of the knotted suture. Based on the results of the study, any subjective impressions of simulation suture strength and knot holding should not negatively impact its use for medical education.

Boot Camp in a Box: Initial Experience with Pretraining Skills Preparation for New Interns.

PURPOSE: In order to increase selected skills at onset of training, we provided newly matched PGY-1 trainees with materials and instructions to practice these skills, as well as the opportunity to share video-recorded performance and receive feedback based on these videos. METHODS: Knot tying and suturing kits, instruments and supplies, and video instructions for task performance were sent to newly matched trainees to our program (n = 10), with instructions to practice 4 tasks (1- and 2-handed knot tying, interrupted and running suturing) until self-assessed comfort with each task was achieved or the 8-week time point before start of training was reached. Each trainee returned a video of each task, which was graded by blinded reviewers for time and errors using an itemized evaluation instrument (12 items for suturing and five items for knot-tying). Feedback (annotations of submitted videos) was provided after grading was completed. Task performance was repeated and reassessed at the time of new intern "Boot Camp" and again 8 weeks after start of training. Performance scores were compared for the 3 time points and with scores of PGY 2-4 residents using ANOVA with posthoc tests. RESULTS: Compliance with instruction for practice and return of video recorded tasks in the months before start of PGY-1 training was high, with only 1 of 10 failing to return knot-tying videos. A significant pattern of performance change (p < 0.05) was observed for all tasks with an initial decrease between the pre-employment practice period and the Boot Camp test followed by an increase to the highest level of performance 2 months after start of training. At that point, scores were not significantly different than those of more senior residents. CONCLUSIONS: A high level of compliance was achieved with requested skills practice and video documentation of performance. We attribute the consistently lower scores on the tasks during Boot Camp tests to higher stress test environment which was apt to be less favorable than having the trainee choose to submit their best possible preresidency video recording of performance in a low-stress situation. Subsequent achievement of significantly higher performance even compared to more senior residents may have been helped by incentivized pretraining practice.